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PREFACE 


The recording of optical imagery is the primary mode of data acquisition in 
many microgravity science experiments, particularly those in the combustion 
science, fluids, and transport disciplines. While the required spatial and 
temporal resolution of the imaging varies from experiment to experiment, there 
is in general a requirement for high speed, detailed optical data. Most 
current spaceflight experiments (developed or under development) requiring 
optical data rely on motion picture film photography to record the data. 
Available space qualified cameras have the ability to meet the high resolution 
and high speed recording requirements of these experiments, but the need for 
large amounts of film has caused serious payload design problems, or even 
constrained the amount of data that could be made available from a flight. In 
order to resolve this problem, the evolutionary development of a high 
resolution, high frame rate video system for microgravity science and 
applications investigators has been initiated at NASA Lewis Research Center. 

It is planned that this system will be used in the future on the Space Shuttle 
and Spacelab, and ultimately on Space Station Freedom. 

The High Resolution, High Frame Rate Video (HHVT) Workshop was held at 
Lewis Research Center on May 11-12, 1988 for the dual purpose of (1) allowing 
potential scientific users to assess the utility of the proposed system for 
monitoring microgravity science experiments and (2) letting technical experts 
from industry recommend improvements to the proposed near-term HHVT system. 
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I NTRODUCT I ON 


The High Resolution, High Frame Rate Video (HHVT) Workshop was held at NASA 
Lewis Research Center on May 11-12, 1988. On the first day of the workshop 
members of the HHVT project team gave presentations on the conceptual design 
of the near-term HHVT system, the state of the art in video technology, the 
results of the Users' Requirements Survey that NASA has conducted previously, 
image enhancement methods, data compression, and data transmission networks. A 
text version of each presenter's vugraphs has been included in this Conference 
Proceedings. The second day of the workshop was reserved for two working 
group discussion sessions, followed by a joint discussion to review the 
findings and recommendations made by the two splinter groups. A summary of 
the comments made by the participants of the Users Working Group and the 
Technical Experts Working Group is Included in this Conference Proceedings 
followed by a list of their findings and recommendations. 
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HHVT WORKSHOP ISSUES 


In planning the topics to be discussed at the HHVT Workshop, the following list 
of issues was generated: 

(1) How realistic are the requirements of the potential users of the HHVT 
system as expressed in their responses to the users' survey? 

(2) Are the requirements driven by the nature of the science and, hence, 
unchangeable? Given that time constants impact frame rate, are the time 
constants of the experiment well understood? Given that spatial dynamics 
impact resolution, are the spatial dynamics of the experiment well understood? 

(3) Which technologically limiting factors become apparent from the user 
survey? Can these imaging requirements be met by efficient and clever 
application of a video system with limited capability? 

(4) The proposed near-term HHVT system represents the best we can do with 
near-term technology. Does its performance capability represent an advancement 
sufficient to justify its development at this time? 

(5) Will the proposed near-term HHVT system be useful to the microgravity 
science experimenters? 

(6) Is the proposed near-term HHVT system performance capability truly 
state of the art? 

(7) Are there anticipated technological developments coming soon that 
would improve the near-term system? 

(8) Are there any identifiable problems with the conceptual design of the 
near-term system? 

(9) Do the users and the technical experts think the overall HHVT 
development project is headed in the right direction? 

All of these topics were discussed at the workshop. 
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RESULTS OF THE USERS' REQUIREMENTS SURVEY 


Robert L. Butcher 
NASA Lewis Research Center 


The objectives of the High Resolution, High Frame Rate Video Technology ( HHVT ) 
Users' Requirements Survey were the following: 

(1) Oocument the requirements of potential users of the HHVT system 

(2) Establish a database relating key videoparameters to HHVT users 

(3) Guide the development of a high resolution, high frame rate video 

system offering high data storage capacity and high data 
transmission rates 

(4) Allow users to compare their requirements to those of other users and 

to state-of-the-art technology 

(5) Allow users to reassess, if necessary, their requirements in light of 

existing and near-term technology 

The Users' Requirements questionnaires were distributed to the following 
potential microgravity HHVT users: 


Data Base 

ID Numbers Potential Microqravity HHVT Users 

100‘s ----- LeRC project scientists of approved flight experiments 

200's LeRC project scientists of ground-based science 

discipline areas which may lead to future microgravity 
fl ight experiments 

300' s Non-LeRC project scientists of ground-based science 

discipline areas which may lead to future microgravity 

flight experiments 

400's ----- Microgravity Science and Applications Division principal 

i nvesti gators 

500' s ----- Microgravity Science and Applications Division 
discipline working groups 

600's Miscellaneous 

700's Private industry 
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The key video parameters solicited in the questionnaire for each experiment or 
ground-based research were 

(1) Spatial resolution (in pixels/frame) 

(2) Frame rate (in frames/sec) 

(3) Gray scale resolution (up to 256 levels) 

(4) Monochrome or color images 

(5) Number of frames to be stored per experiment run 

(6) Number of frames to be stored per flight 

(7) Downlinking requirements 

The first HHVT Users' Requirements Survey was issued in December 1986. Sixty- 
eight questionnaires were mailed. Seventeen of the thirty-nine completed 
questionnaires indicated video imaging requirements. 

The same questionnaire was issued to 119 different investigators in May 1987. 
Eighteen of the twenty-eight completed questionnaires that were received 
indicated video imaging requirements. 

In October 1987 the original questionnaire was mailed to fifteen more 
investigators. Both of the two completed questionnaires received indicated 
video imaging requirements. In November 1987 a different questionnaire that 
was written by engineers working on the HHVT project at Langley Research 
Center was mailed to all the Investigators who had responded to the original 
questionnaire. This second questionnaire sought detailed information on the 
experimental images, image enhancement requirements, and real-time monitoring 
requirements. Fourteen investigators responded to the second questionnaire. 
Thirteen of these indicated a need for image enhancement. 

The results of the HHVT Users' Requirements Survey have been tabulated and 
appear in Appendix I - HHVT Video Requirements Survey; Appendix II - HHVT 
Downlink Requirements; and Appendix III - Summary of Image-Processing 
Requirements appended to this report. 

The formula used to calculate the downlink data rate requirement is as follows: 

Data rate (bytes/sec) = [Spatial resolution (in pixels/frame)] 

x [log 2 (gray scale resolution in bits/pixel)] 
x [1 byte/8 bits] x [downlink frames/run] 
x [1 downlink time in sec] x [Y] 

where 

Y = 1 (for monochrome), 3 (for color) 
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For example, User No. 102 In the HHVT Users' Requirements database has the 
following requirements for the Solid Surface Combustion Experiment: 

Spatial resolution = 5000 x 2500 pixels = 1.25xl0 7 
Gray scale = 256 

Frames per run to be downlinked = 23 040 

Period during which downlinking must occur = 12 hr 

Color images 

Data rate (bytes/sec) = [1.25xl0 7 pixels/frame] 

x [8 bits/pixel] x [1 byte/8 bits] 
x [23 040 frames/run] x [1/12 hr] x [1 hr/3600 sec] 
x [3] 

= 2.00xl0 7 bytes/sec 

Currently, 37 potential microgravity HHVT users have submitted diverse video 
requirements. The requirements for spatial resolution range from 4100 pixels/ 
frame to 100 000 000 pixels/frame (which is greater than the resolution 
achievable with 16 mm film). Framing rate requirements vary from four (4) per 
hour to 1 000 000 frames/sec. The results of the Users' Requirements Survey 
are as indicated in the attached appendixes (see pp. 6-17). The diversity of 
these requirements Indicates a need for developing a video system with great 
flexibility. Further consideration has to be given to how the results of this 
survey correspond to plans being made for video systems to be installed onboard 
the Space Shuttle and Space Station Freedom. 
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APPENDIX I 

HHVT VIDEO REQUIREMENTS SURVEY 
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DOWNLINK REQUIREMENT 
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APPENDIX III 

SUMMARY OF IMAGE-PROCESSING REQUIREMENTS 


(SURVEY RESPONSE TO LANGLEY RESEARCH CENTER REQUEST TO OBTAIN 
DETAILED IMAGE INFORMATION, IMAGE ENHANCEMENT REQUIREMENTS, 
AND REAL-TIME MONITORING REQUIREMENTS) 
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SUMMARY OF IMAGE-PROCESSING REQUIREMENTS 
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N91-14576 


DATA TRANSMISSION NETWORKS 

Robe r t A I exov i ch 
Analex Corporation 
NASA Lewis Research Center 


The High Resolution, High Frame Rate Video Technology (HHVT) project engineers 
wrote a task order to Analex Corporation to investigate data compression 
techniques that could be applied to the HHVT system, and both existing and 
planned downlink/uplink capabilities of the Space Shuttle and Space Station 
Freedom. Specifically, Analex Corporation was directed to do the following: 

(1) Investigate signal channel availability and determine both the maximum 

possible data rate and the average data rate 

(2) Identify time blocks for HHVT video transmission assuming time sharing 

and interruptions in the communications links 

(3) Determine the bit error rates to be expected 

(4) Define the transmit and receive Interfaces 

The formula used to determine the bit rate is as follows: 

Bit rate = (HRXVR)(IR)(FR)(C)(V) 


where 

HR horizontal resolution In pixels 

VR vertical resolution In pixels 

IR Intensity resolution In bits 

FR frame rate In frames/sec 

C 3 (for color), 1 (for monochrome) 

V number of views 

For example, with HR = VR = 512 pixels, IR = 8 bits (which corresponds to 
256 level resolution), FR = 30 frames/sec, C = 3 for color, and V = 1, 

Bit rate = 512 x 512 x 8 x 30 x 3 x 1 = 190 Mbits/sec 

There are several constraints on the amount of data that can be downlinked at 
any given time. These constraints include the maximum possible data 
downlinking rate, geometric coverage, antenna blockage, data acquisition, and 
the fact that the data transmission network is a multiuser shared resource. 

A summary chart of the data transmission capabilities for TRSS, the Space 
Shuttle, Space Station Freedom, Spacelab, and USLab is attached. 
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SCHEDULE OF RELATED SYSTEMS 


1 j 1989 1 

90 

91 

92 

93 

94 

95 

96 

97 

98 

lJL 

2000 

GSTDN 

E3I 











Ml 

TDRS(41 W) 


mmmmi 

I'M) I i i ; 

Hi' LLill 

i ■ . m ; 1 i 

[LLLlIlL 





1 


TDRSU71W) 

■ " 






ATORSS 







BZ — 

STS 

SLS 

i i 

m 

SB 

g 


jH 






1 

SS PHASEJ 


m 





. >i .i 


■MM 

il 

USLAB 








i- — J 

JEM 


3 

1 






i 

COLUMBIA 









WBSB P— 1 


TDRSS SPACE SEGMENT 



19 


OPIGINAL 'AGE IS 
0." POOR 0UAL5TY 



































SHUTTLE ORBITAL COMMUNICATIONS LINKS 



S BANO 

• PM UPLINK (3?kH»tl 

• PM OOWNIINK 196 kbptl 


KU BAND 

• rM UPLINK 177 hbp, « 1 Mb#,l 
« PMDOWNIINK UlMbM » iSOMbtil 


NOT AVAILABLE 
PAST 1989 


S BANO 

• fM UPLINK 17? bhptl 

VOICE (7 > 37 hfefHl COMMAND*; 6 bp« 
17 bbpt INFORMATION ENCODED! ANO 
1.1 kbpi SYNCHRONIZED INTERLEAVED 



• rM DOWNLINK 119? the* I VOICE 
17 * 37 bbp*| ANO 1 J§ kb*. ORRITER 
reM TELEMETRY with intfrleaved 
frequency fayload data 

1(4 kbpi MAKIMUMI 


torss 

CROUNO STATION 


• FM DOWN1 INK 

t im F mi a nr p vyiof flflfin fayioao d a r a 
I a f j a ( or, on null r ai i t r i r vision fmimi* 


cnoiiNO Sfatt inAfKiNr, AND 
OATA NETYVOnK STATION 


Rrtunuf 0 DATA. TO 4 0 Mil. on SO Mbi»» 


ORIGINAL PAGE IS 
OF POOR QUALITY 



SPACE STATION POWER MODULE AND TRUSS 







BITS/ORBIT Log (bits/orbit) 


USER REQUIREMENTS 


TDRSS SUPPORT 


13 “ 

14 - 







B 




13 - 






■ 

m 

■ 




12 -a 





... 

■ 

■ 

■ 

■ 

■ 




■ 

1C 

11 - 

■ 


# 

■ 


■ 

1 

■ 





10 - 


■ 

■ 

m 

m 

■ 

■ 

1 P 


■ 

MAX RA1 

E SOOmb 

DS 

9 - 



m 

■ 

■ 


m 

■ 

■ 

TDRSS 1 
TDRSS 3 

990 - 1 
995 - 2 

995 

DOO 

0 - 

■ 

■ 

■ 

■ 

6 









9 11 13 15 

BIT RATE Log (bits/second) 


USER REQUIREMENTS 


SHUTTLE SUPPORT 


13 “ 

14 - 











4J 

1 1 - 





■ 

■ 

n 




L. 

O 

N 

to 

12 - 



I 

I 


■ 

■ 

■ 

■ 

■ 

b 



■ 

CT1 

O 

-J 

h- 11 - 

■ 


\§ 

k 

m 

j ■ If ' 

■ 

■ 

E 




H 

CQ 

CL 

□ 

£ 10 - 


■ 

■ 

■ 

■ 

i 

i 

! 

■ 

■ 

■ p 


■ 

MAX RA 

TE 48mb 

3S 

(D , 

9 - 




i 

m 

i 

-m- 

m 

■ 

■ 


1 HOR 
5 HDR 

USER + 
USERS 

4IX 

8 - 

■ 

■ 

■ 

■ j 

— 




I 





9 11 13 15 

BIT RATE Log (bits/second) 


24 




USER REQUIREMENTS 
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N91 - 14577 


DATA COMPRESSION APPLIED TO HHVT 

William K , Thompson 
Analex Corporation 
NASA Lewis Research Center 


The H 1 qh Resolution, High Frame Rate Video Technology (HHVT) project engineers 
wrote a task order to Analex Corporation to study data compression teehniques 
that could be applied to the HHVT system. Specifically, the goals of the HHVT 
data compression study are to accomplish the following: 

(1) Determine the downlink capabilities of the Space Shuttle and Space 
Station Freedom to support HHVT data (i.e., determine the maximum 
data rates and link availability) 


(2) Determine current and projected capabilities of high speed storage 

media to support HHVT data by determining their maximum data 
acquisition/transmission rates and volumes 

(3) Identify which experiments in the HHVT Users' Requirements database 

need data compression, based on the experiments' imaging requirements 

(4) Select the best data compression technique for each of these users by 

identifying a technique that provides compression but minimizes 
di s tort ion 

(5) Investigate state-of-the-art technologies for possible Implementation 

of selected data compression techniques 


Data compression will be needed because of the high data rates and large 
volumes of data that will result from the use of digitized video onboard the 
Space Shuttle and Space Station Freedom. For example, the data rates and 
volumes stemming from the use of standard RGB video and HHVT RGB video are 
compared in the following table: 


Standard HHVT 

RGB Video RGB Video 


Spatial resolution, pixels 
Color components 
Quantization, levels 
(8 bi ts) /component 
Frame rate, frames/sec 
Duration, frames 


512 x 512 

3 (Red, green, blue) 


1024 x 1024 
3 (Red, green, blue) 


256 

30 (interlaced scan) 
10 000 


256 

1000 (no Interlacing) 
10 000 


Resultant data rate 190 Mbits/sec 

Required storage volume. Gbits 63 


26 Gbits/sec 
520 
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Existing high speed data storage systems and those expected to be commercially 
available within the next few years cannot support the very high data rates 
that are generated by some potential users of the HHVT system. Many of these 
experiments will require data compression during their data acquisition cycles 
just to be compatible with existing data storage devices. 

The downlink capabilities of the Space Shuttle and Space Station Freedom cannot 
support the large volumes of HHVT data due to limited link availability For 
example, the link will be available to the USLab on Space Station Freedom for 
74 min/orbi t. With a maximum downlink rate of 75 Mbits/sec, the maximum volume 
of data that can be downlinked per orbit is 333 Gbits. Assuming the data must 
be downlinked before the next orbit, data compression needs to be applied after 
the data has been acquired but prior to downlink. Nonreal-time processing 
could be used since the data is not necessarily required immediately after it 
has been acquired. 
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II. THEORETICAL APPROACHES TO DATA COMPRESSION 


• FIRST GENERATION APPROACH 

• BASED ON INFORMATION THEORY AND STATISTICS 

• IMAGES TREATED AS 2-D OR 3-D RANDOM FIELDS (V.H.T) 

• EXPLOITS CORRELATION OF A PEL WITH ITS SPATIAL AND TEMPORAL NEIGHBORS 




• DATA COMPRESSION BY STATISTICAL REDUNDAN CY REDUCTION 


II. THEORETICAL APPROACHES TO DATA COMPRESSION 


• FIRST GENERATION APPROACH 

■ CORRELATION BETWEEN PELS DECREASES WITH INCREASING SPATIAL 
OR TEMPORAL DISTANCE BETWEEN PELS 

- CAUSAL APPROACHES ONLY REDUCE CORRELATION OF PEL 14 WITH 
PELS 1-13 SINCE PELS 1-13 ARE SCANNED PRIOR TO PEL 14 

- INTRAFRAME APPROACHES ONLY REDUCE CORRELATION OF PEL 14 
WITH OTHER PELS IN F2 

- ADAPTIVE TECHNIQUES VARY THEIR CODING EFFICIENCY BASED 
ON SCENE ACTIVITY 
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H. THEORETICAL APPROACHES TO DATA COMPRESSION 


SECOND GENERATION APPROACH 

• EXTENDS INFORMATION THEORY TO INCLUDE VISION AND IMAGE GATHERING MODELS 

• MOST IMPORTANT INFORMATION IS EXTRACTED FROM THE IMAGE 

• INSPIRED BY PROPERTIES OF THE HUMAN VISUAL SYSTEM (HVS) 


PERCEIVED 

INTENSITY 



BRIGHTNESS 



1. WEBER'S LAW 2. LATERAL INHIBITION 


3. BAND PASS SFR 


4. LOW PASS TEMPORAL FREQUENCY RESPONSE, INTEGRATING FOR .1 SEC 

5. CONTOUR PERCEPTION 

6. MOTION DEPENDENT SPATIAL RESPONSE 

7. DIRECTIONAL SENSITIVITY - MORE SENSITIVE TO VERTICAL AND HORIZONTAL 
DATA COMPRESSION BY FEATURE EXTRACTION 


II. THEORETICAL APPROACHES TO DATA COMPRESSION 

OPPORTUNITIES FOR DATA COMPRESSION IN DATA PATH 



DOWNLINK 


DATA COMPRESSION FOR LONG TERM HHVT SYSTEM MAY BE PERFORMED AT 
MULTIPLE LOCATIONS IN THE DATA PATH 


30 





II. THEORETICAL APPROACHES TO DATA COMPRESSION 


COLOR 1 

• COMPOSITE SIGNAL CODING VS. COMPONENT CODING 

• COMPONENT CODING HAS GIVEN BEST RESULTS IN THE LITERATURE 

• BEST RESULTS WHEN YIQ COMPONENTS ARE USED INSTEAD OF RGB 

Y = LUMINANCE SIGNAL l,Q = CHROMINANCE SIGNALS 

. RGB TO YIQ CONVERSION IS STRAIGHTFORWARD AND LINEAR 

• YIQ COMPONENTS ARE NEARLY UNCORRELATED, RGB ARE HIGHLY 
CORRELATED 


III. PERFORMANCE CRITERIA 


PERFORMANCE IS HIGH CODING EFFICIENCY WITH HIGH FIDELITY 


WAYS OF EVALUATING CODING EFFICIENY 


1 . 

2 . 

3. 


ACHIEVABLE COMPRESSION RATIO AT A GIVEN FIDELITY CRITERION 
UNCORRELATED DATA (FIRST GENERATION) 

MINIMAL DATA REPRESENTATION OF DESIRED FEATURE (SECOND GENERATION) 


WAYS OF EVALUATING FIDELITY 


1. SUBJECTIVE OBSERVER 

IS THE OUTPUT IMAGE AESTHETICALLY PLEASING? 


2. QUANTITATIVE 

IS THE ERROR MATHEMATICALLY MINIMIZED? 

- MEAN SQUARE E = (f - f ) 2 

- MAGNITUDE E = |f - f| 


3 . 


FEATURE SPECIFIC 

ARE IMPORTANT FEATURES IN THE OUTPUT 

• EDGE AND CONTOUR INFORMATION 

• LUMINANCE / REFLECTANCE INFORMATION 


PROPERLY REPRESENTED? 

• COLOR INFORMATION 

• MOTION SEQUENCES 
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IV. COMPRESSION ALGORITHMS FOR HHVT 


• QUANTIFICATION OF USER REQUIREMENTS 

• EACH USSR'S VIDEO REQUIREMENTS CHARACTERIZED IN TERMS OF SIX 
PARAMETERS AND THEIR IMPORTANCE TO THE USER'S REQUIREMENTS 

• SCORED FROM 0 (NOT IMPORTANT) TO 5 (VERY IMPORTANT) 

• PARAMETERS CONSIDERED 

1. SPATIAL INFORMATION 

- PLACEMENT AND SHARPNESS OF EDGES, CONTOURS 

- TEXTURE INFORMATION 

2. TEMPORAL INFORMATION 

-ACCURATE REPRESENTATION OF MOTION: SPEED AND DIRECTION 

3. AESTHETIC APPEARANCE 

- SUBJECTIVE EVALUATION BY HUMAN OBSERVER 

4. LUMINANCE / REFLECTANCE INFORMATION 

- CONTRAST 

- INTENSITY 

5. SPECTRAL INFORMATION 

- COLOR 

6. IMAGE DYNAMICS 

- ADAPTABILITY 

- NEED FOR COMPRESSION ALGORITHM TO VARY PERFORMANCE 
BASED ON SCENE ACTIVITY 


IV. COMPRESSION ALGORITHMS FOR HHVT 


QUANTIFICATION OF COMPRESSION TECHNIQUE PERFORMANCE 

- EACH TECHNIQUE SCORED FOR EACH PARAMTER BASED ON HOW WELL IT 
PRESERVES THE PARAMETER WITHOUT INTRODUCING DEGRADATION 

EACH TECHNIQUE SCORED AT COMPRESSION RATIO REQUIRED BY EXPERIMENT 

0 = POOR PERFORMANCE 
5 = EXCELLENT PERFORMANCE 

EVALUATING A GIVEN TECHNIQUE FOR A GIVEN EXPERIMENT 

- FORM A REQUIREMENTS VECTOR (SIX ELEMENTS) 

- FORM A PERFORMANCE VECTOR (SIX ELEMENTS) 

- TAKE A DOT PRODUCT AND NORMALIZE 


EXAMPLE: 


[533201 ] 
MAX = 70 


5 

2 

2 

1 

1 

5 


= 5 ( 5 ) + 3(2) + 3(2) + 2(1) + 0(1) + 1(5) = 44 

44/70 = 63% 

SCORE 


THOSE COMBINATIONS RECEIVING HIGHEST SCORES WILL BE EVALUATED 
FURTHER 
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IV. COMPRESSION ALGORITHMS FOR HHVT 


. OTHER CONSIDERATIONS OF IMPORTANCE 

1. TECHNIQUE COMPLEXITY 

- SPEED 

- WEIGHT 
-COST 

- DEVELOPMENT TIME 

- POWER 

2. SUSCEPTIBILITY TO CHANNEL ERRORS 

- AFFECT ON IMPORTANT PARAMETERS 

- LOCAL EFFECTS VS. AVERAGED EFFECTS 

- RESTORABILITY 


V DATA COMPRESSION OVERVIEW 


LOSSLESS CODING METHODS 


• IMAGE IS FULLY RECOVERABLE GIVEN ERROR-FREE TRANSMISSION 
EXAMPLES: 

RICE ALGORITHMS (CODED DIFFERENCES) 

RUN -LENGTH CODING 
BIT PLANE CODING 
CONDITIONAL REPLENISHMENT 
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V. DATA COMPRESSION OVERVIFW 


• LOSSY PREDICTIVE CODING 

- IMAGES ENCODED BY QUANTIZING THE ERROR BETWEEN THE PREDICTED VALUE 
OF A SUBPICTURE AND THE ACTUAL VALUE 

- SUBPICTURES MAY BE INDIVIDUAL PELS OR VECTORS 


INPUT DATA 




* 


QUANTIZER 


CHANNEL 


PREDICTOR 
WITH DELAY 


Kr 


- EXAMPLES 

- DPCM, VECTOR DPCM 

- DELTA MODULATION 


- MADE ADAPTIVE BY VARYING QUANTIZATION 
SCENE ACTIVITY 


OR PREDICTION PARAMETERS WITH 


V. DATA COMPRESSION OVERVIFW 


• TRANSFORM CODING 

- LINEAR TRANSFORM (ORTHOGONAL BASIS SET) DESCRIBES SPATIAL "FREQUENCY" 
DOMAIN BEHAVIOR OF IMAGE 


INPUT 

BLOCK 


CHANNEL 



OUTPUT 

BLOCK 


LOW MAGNITUDE COEFFICIENTS IN "FREQUENCY DOMAIN" ARE UNDER QUANTIZED 
OR OMITTED DURING RESAMPLING AND REQUANTIZATION 


- THRESHOLD SAMPLING: SELECTIVE TRANSMISSION BASED ON COEFFICIENT 
MAGNITUDE (ADAPTIVE) 


- ZONAL SAMPLING: FIXED SAMPLING AND QUANTIZATION PATTERN DETERMINED 
A PRIORI FROM ANTICIPATED IMAGE STATISTICS (NON-ADAPTIVE) 

- ZONAL SAMPLING IS OFTEN MADE "CLASS-ADAPTIVE” 


- EXAMPLES OF COMMONLY USED BASIS FUNCTIONS 

- KARHUNEN-LOEVE: OPTIMAL BUT VERYDIFFICULT TO IMPLEMENT 

- COSINE, FOURIER, SLANT, HADAMARD, HAAR: SUBOPTIMAL BUT EASIER 
TO IMPLEMENT 
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V DATA COMPRESSION OVERVIEW 


VECTOR QUANTIZATION 

- PARTITION THE IMAGE INTO BLOCKS TO BE TREATED AS VECTORS 


input vector 



- CODEBOOK DESIGNED A PRIORI WITH TRAINING DATA OR ADAPTIVELY TO 
MINIMIZE MEAN-SQUARE ERROR 

- SEARCH ALGORITHMS FOR BEST FIT IN CODEBOOK CAN BE FULL (OPTIMAL) 

OR TREE-STRUCTURED (SUBOPTIMAL, BUT FASTER) 

- DECODER IS A SIMPLE LOOK-UP TABLE 

- LARGE COD6BOOKS GIVE BETTER OUTPUT, BUT COMPLEXITY RISES SHARPLY WITH 
CODEBOOK SIZE 


V DATA COMPRESSION OVERVIEW 


• FEATURE EXTRACTION 

• DETERMINE A PRIORI WHAT INFORMATION IS IMPORTANT IN THE IMAGE 

• EXTRACT THIS INFORMATION UP FRONT AND DISCARD THE REST 

• EXAMPLES: 

- CONTOUR CODING 

- SYNTHETIC HIGHS 

- IDS 

- DIRECTIONAL DECOMPOSITION 

- CONTOUR TEXTURE CODING 

. THESE TECHNIQUES OFFER THE HIGHEST COMPRESSION RATIOS 
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DATA COMPRESSION OVERVIEW 


CATEGORY 

LOSSLESS 

LOSSY 

PREDICTIVE 

TRANSFORM 

VQ 

FEATURE 

Extraction 

COMPRESSION 

FACTOR 

2 

8 

16 

20 

100 

LEAD 

TIME 

NOW 

NOW 

NOW 

1 YR 

5 YRS 

COMPLEXITY 

LOW 

LOW 

TO 

MODERATE 

MODERATE 

TO 

HIGH 

MODERATE 

TO 

HIGH 

HIGH 

DISTORTION 

EFFECTS 

NONE 

QUANTI- 
ZATION 
NOISE. 
LOSS OF 
DETAIL 

LOSS OF 
1 DETAIL. 
BLOCKING 

BLOCKING. 

ARTIFACTS 

LOSS OF 
UNEXTRACTED 
INFORMATION 

SENSITIVITY 
TO CHANNEL 
ERRORS 

MODERATE 

TO 

HIGH 

HIGH 

MODERATE 

MODERATE 

VERY 

HIGH 

CHANNEL 

DELAY 

LOW 

LON 

MODERATE 

MODERATE 

LOW 

FLEXIBILITY 

HIGH 

MODERATE 

TO 

HIGH 

HIGH 

HIGH 

LOW 
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N91 - 14578 


STATE OF THE ART IN VIDEO SYSTEM PERFORMANCE 

Michael J. Lewis 
NASA Lewis Research Center 


The closed circuit television (CCTV) system that is onboard the Space Shuttle 
has the following capabilities: 

Camera 


- based on a silicon intensified target (SIT) image tube 

- has approximately 300 TV lines of horizontal resolution 

- the full frame rate is 30/sec for monochrome images, 

20 frames/sec for field sequential color format 

- mates to a commandable pan and tilt unit 

- has a commandable zoom lens assembly 

- has operating settings for the iris, gamma, shading correction, 
etc . 

Video signal switching and routing unit (VSU) 

- accepts up to 14 input video signals, including up to 3 from 
cargo bay payloads and up to 4 from the crew compartment 
(which includes the middeck) 

- routes video signals to 12 output ports, including two TV 
monitors, the Space Shuttle video tape recorder (VTR) , and one KU 
or S band downlink channel 

- allows two camera views to be multiplexed onto one composite 
signal which can then be recorded, displayed, or downlinked 

- provides time tags to the video signals 

Space Shuttle video tape recorder (VTR) 

- records monochrome or color format video and audio from the VSU 

- the playback video signal can be routed to the VSU or directly to 
the No. 2 TV monitor 

- one 3/4 in. cassette can record for 30 min 

Useful as it is, the Space Shuttle CCTV system is inadequate for use with many 
experiments that require video imaging. Some of these shortcomings are 

(1) The CCTV camera cannot frame at more than 30 frames/sec. 

(2) The camera does not have high resolution. 
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(3) The CCTV system lacks the ability to trade off resolution and frame 
rate in order to accommodate experiments with different imaging requirements. 

(4) The camera lens has to be changed in order to switch from monochrome 
to color imaging. 

(5) The CCTV camera captures color images in field sequential format 
which produces an image flicker effect on the Space Shuttle monitors. Only 
monochrome images can be displayed on Shuttle monitors because of this 
flickering. Also, the image data in field sequential format must be converted 
to NTSC format before it can be distributed on ground. 

(6) The video tape recorder's 30-min maximum recording time is 
insufficient for experiments that require more than 30 min to run their course. 

NASA does have a contract underway to build a 3-chip CCD color /monochrome 
camera using NTSC format to replace the existing SIT Shuttle cameras. 

In order to assess the state of the art in video technology and data storage 
systems, NASA Lewis Research Center has been conducting a survey of 
manufacturers. The HHVT project engineers have searched the technical 
literature on state-of-the-art imaging components and data handling systems, 
as well as meeting with vendors and/or discussing their products via the 
telephone. The project engineers have investigated the state of the art in 
commercially available high frame rate video systems, image sensors and 
cameras, high speed data storage systems, and mass storage data systems. The 
database generated will be updated throughout the duration of the HHVT 
development project. 

The performance of state-of-the-art solid state cameras and image sensors, 
video recording systems, data transmission devices, and data storage systems 
versus users' requirements are shown graphically on the attached charts (see 
Figs. 1 to 4). Figure 3 shows maximum data transfer rates versus users' 
requirements. In order to plot the transfer rate in terms of the highest 
possible frame rate versus resolution for a given data storage system, 1 byte/ 
pixel was assumed to be sufficient. It is important to note that this graph 
assumes that the overall video system is not limited by the capabilities of the 
image sensor. 

In comparing the various available data acquisition/transmission/storage 
systems, the dynamic random access memory (RAM) has numerous advantages. These 
advantages are 

(1) Very high speed data acquisition 

(2) Slower speed random access transfers 

(3) Error correction 

(4) Varying input data rates are acceptable 

(5) Compatiblity with various control busses 

(6) Compactness 
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One disadvantage of the dynamic RAM is that it has a relatively low storage 
capacity for extended high frame rate imaging. 

The advantages of random file access systems such as PDT, NPDT, and optical 
disks are 

(1) Large data capacity for extended imaging times 

(2) Compatibility with various control busses 

(3) Replaceable disks for greater storage capacity 

Their disadvantages include 

(1) Relatively slow data acquisition rates for high frame rate imaging 

(2) Little or no error correction 

(3) Questionable ruggedi zation of disks and disk drives 

(4) Large volume requirements for most magnetic disk systems 

Magnetic tape recorders have numerous advantages including 

(1) Very large data capacity for extended imaging times 

(2) Variable data acquisition/transmission rates for recording and 

pi ayback 

(3) Relatively high data transfer rates 

(4) Error correction 

(5) Replaceable tape cartridges for greater storage capacity 

(6) Ruggedness 

(7) Compactness 

Their disadvantages are 

(1) Relatively slow data transfer rates for direct recording of high frame 

rate images 

(2) Incompatibility with various control busses 

(3) Possible file access delay 

Sixteen millimeter film has been used reliably in the past because there are 
numerous film cameras which allow very high frame rates with high resolution 
and the cameras have been ruggedized for flight. But their disadvantages are 
that images can only be acquired for very short durations at high frame rates 
because the film magazines run out so fast, and longer imaging times even at 
moderate frame rates require large amounts of film to be used. Also, 
astronauts are needed to replace the film magazines, adding extra work to 
their already busy schedules. On Space Station Freedom replenishment of film 
magazines will only occur every 3 months. 
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The conclusions that have been drawn from this survey of state-of-the-art video 
technology and memory systems are as follows: 

(1) Current imaging systems and solid state sensors fall short of 
achieving a major portion of the users' requirements. 

(2) Custom CID or CCD sensors could accommodate more of the users' 
requirements . 

. <3) Some data storage systems have the potential to satisfy a major 
portion of the users' requirements. 
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SOI Tn STATE TMflry SFNSORS AND C AMERA DATA 


0 TEKTRONICS 
TK128PR 


0 RET I CON 
RA2566N 


0 GENERAL ELECTRIC 
CID512 CAMERA 


0 GENERAL ELECTRIC 
CID 25 6E SENSOR 


0 KODAK VIDEK 
MEGA PLUS 


0 TEKTRONICS 
TK20A8M 


- RESOLUTION - 6A X 128 PIXELS 

- FRAME RATE - > 5000 FRAMES/SEC. 

- PIXEL RATE - 5 MHz (MAX. ) /CHANNEL 

- VIDEO CHANNELS - 6A 

- RESOLUTION - 256 X 256 PIXELS 

- FRAME RATE - 500 FRAMES/SEC. 

-PIXEL RATE - 5 MHz/CHANNEL 

- VIDEO CHANNELS - 8 

- RESOLUTION - 256 X 256 PIXELS 

- FRAME RATE - 500 FRAMES/SEC. (SKIP SCAN) 

- PIXEL RATE - 7.8 MHZ 

- VIDEO CHANNELS - 1 

- RESOLUTION - 256 X 256 

- FRAME RATE - 150 FRAMES/SEC. 

- PIXEL RATE - 2.5 MHz/CHANNEL 

- VIDEO CHANNELS - A 

- RESOLUTION - 1320 X 1032 

- FRAME RATE - 10 FRAMES/SEC. 

- PIXEL RATE - 1A MHz (MAX) 

- VIDEO CHANNELS - 1 


- RESOLUTION - 20A8 X 20A8 

- FRAME RATE - A FRAMES/SEC. 

- PIXEL RATE - 1 MHZ (MAX) 

- VIDEO CHANNELS - 1 
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PRFSFNTLY AV/ATI ABLE VTOFO SYSTEM Dftlfl 


0 SPIN PHYSICS 
SP 2000 


- FULL FRAME RATE - 2000, 1000, 500, 200, 60 

- SENSOR - MOS ARRAY 

- RESOLUTION - 192 X 240 PIXELS 

- 32 PARALLEL CHANNELS 

- PIXEL RATE - 3.0 Wz 


- RECORDER - FULL RESOLUTION 

- 64 GRAY LEVELS 

- 34 TRACKS 

- 1/2" HIGH DENSITY CASSETTE 


0 NAC HVRB-200SS - FULL FRANE RATE - 200, 60 

- SENSOR - MOS CCD ARRAY 

- RESOLUTION - 244 X 280 PIXELS 

- RECORDER - 200 + HORZ. LINES 

- 1/2” VHS TAPE 


0 RSI 70 CAMERA & - FULL FRAME RATE - 30 

VHS VCR - SENSOR - CCD, CID, PHOTO-CAPACITOR, TUBE 

- RESOLUTION - 483 X 441 PIXELS 
- RECORDER - FULL FRAME RATE - 30 

- RESOLUTION - 483 X 320 


0 16W FILM - RAR 2498 FILM 

- ASA 125 

- FULL FRAME RATE - NO LIMIT 

- RESOLUTION - 1920 X 2400 PIXELS 
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hfnfrai COMPONENT DATA F DR DATA TRANSF E R RAT E AND STORAG E 


0 DYNAMIC RAM 


0 NON-PARALLEL MAG. 
DISK TRANSFER 


0 MAG. TAPE REC. 


0 PARALLEL MAG. DISK 
TRANSFER 


ZITEL 93+ SPLIT 

TRANSFER RATE - 690 MBYTES/SEC. 

STORAGE CAPACITY - 512 MBYTES 
DATA WORD SIZE - 256 PARALLEL CHANNELS 
HIGH SPEED SEQUENTIAL BLOCK TRANSFER 
SLOWER SPEED RANDOM WORD READ/WRITE 
SIZE - 19" X 19" X 21.2" 

- AVAILABILITY - LATE ’88 


- RECOGNITION CONCEPTS INC. VISISTORE 

- TRANSFER RATE - 79 MBYTES/SEC. (SUSTAINTED) 

- STORAGE CAPACITY - 25 GBYTES 

- DATA WORD SIZE - 69 PARALLEL CHANNELS 

- RANDOM FILE READ/WRITE 

- SIZE - 60" X 19” X 27" (2) 

- AVAILABILITY - MID TO LATE '88 


- MIL. STANDARD 21-79AS 

- TRANSFER RATE - 30 MBYTES/SEC. 

- STORAGE CAPACITY - 99 GBYTES 

- DATA WORD SIZE - 8 PARALLEL CHANNELS 

- SEQUENTIAL DATA READ/VKITE 

- SIZE -170" X 16" X 16" 

- AVAILABILITY - MID '89 


- APPLIED MEMORY TECHNOLOGY MODEL 8300 

- TRANSFER RATE - 7.8 MBYTES/SEC. (SUSTAINED) 

- STORAGE CAPACITY -1.0 GBYTES 

- DATA WORD SIZE - 16 PARALLEL CHANNELS 

- RANDOM FILE READ/WRITE 

- SIZE - 8.75" X 17" X 26" 

- AVAILABILITY - LATE '88 
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GENERAL COMPONENT DATA FOR DATA TRANSFER R ATE AMD STORAGE 

(CONTINUED) 


0 OPTICAL DISK - KODAK 

- TRANSFER RATE - 1 MBYTES/SEC. * 

- STORAGE CAPACITY - 6.8 GBYTES 

- DATA WORD SIZE - 16 PARALLEL CHANNELS 

- RANDOM FILE READ/WRITE 

- SIZE - 

- AVAILABILITY - 


0 SHUTTLE KU - VIA TDRSS 

BAND DOWNLINK - TRANSFER RATE - 6.0 MBYTES/SEC. ** 

- FULL USE OF SHUTTLE HIGH SPEED DIGITAL DATA 
CHANNEL 


* INTERFACE LIMITED VIA SCSI 
** DATA RATE FROM WHITE SANDS 
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ADVANCED TECHNOLOGY DEVELOPMENT FOR IMAGE GATHERING, 

COD I NG , AND PROCESS I NG 

Friedrich 0. Huck 
NASA Langley Research Center 


SUMMARY 

Our research activities consist of three overlapping areas. 1) Information theory and optimal 
filtering are extended rigorously to visual information acquisition and processing. The goal is to 
provide a comprehensive methodology for quantitatively assessing the end-to-end performance of 
image gathering, coding, and processing. Information theory allows us to establish upper limits on 
the visual information that can be acquired within given constraints. Optimal mtering allows j* to 
establish upper limits on the performance that can be attained for specific tasks, even if these tasks 
require adaptive or interactive processing. 2) Focal-plane processing techniques and technology 
are developed to combine effectively image gathering with coding. The emph^isisonlo^le 
vision processing akin to the retinal processing in human vision. Our approach includes the fam 
lateral inhibition, the new intensity-dependent spatial summation, and para lei sensing/proce g 
networks. 3) A breadboard adaptive image-coding system is being assembled. This system will 
be used to develop and evaluate a number of advanced image-coding technologies and techniques 
as well as to research the concept of adaptive image coding. The idea of adaptive image coding 
is to use knowledge-based supervision to autonomously select the appropriate data compressio 
scheme based on (a) the properties of the acquired image data (and changes in t^se properties), 
(b) the visual information required by the investigator, and (c) the available channel capacity T 
system would have continual access to the investigators’ knowledge of what is sipuficant. Under the 
investigators’ supervision, it could adaptively select image-coding and feature-classification schemes 
based on properties of the target such as spatial structure, texture, and spectral reflectance. 


1. INFORMATION THEORY AND OPTIMAL FILTERING 

The performance of (digital) image-gathering systems is constrained by the spatial-frequency 
response of optical apertures, the sampling passband of photon-detection mechanisms, and the 
noise generated by photon detection and analog-to-digital conversion. Biophysical limitations 
have imposed similar constraints on natural vision. Visual information is inevitably lost m bo 
image gathering and low-level vision by aliasing, blurring, and noise^ Therefore, it is no longer 
permissible to assume sufficient sampling as Shannon and Wiener could do in their classical wo , 
respectively, on communication theory and optimal filtering for time-varying signals. Nevertheless 
the digital-processing algorithms (for image restoration, edge enhancement, etc.) found in 1 1 
currently prevailing literature assume sufficient sampling, whereas image-gathering systems are 
ordinarily designed to permit considerable insufficient sampling. This fundamental difference 
between assumption and reality has caused unnecessary limitations in the performance of digital 
image gathering, coding, and processing. It also has led to unreliable conclusions about tie 
correct design of image-gathering systems for visual information processing (as opposed to image 
reconstruction without processing, e.g„ commercial TV) and about the actual performance of image- 
coding schemes for tasks which involve digital image processing. 

Our analyses so far have shown that the combined process of image gathering and optimal 
processing (see fig. 1) can be treated as a communication channel if (and only if) the image- 
gathering degradations are correctly accounted for. Correctly restored images gam significantly in 
fidelity (similarity to target), resolution (minimum discernible detail), sharpness (contrast 
large areas), and clarity (absence of visible artifacts). These improvements in visual quality are 
obtained solely by the correct end-to-end optimization without increase in either data transn 
or processing. These results have encouraged us to extend our analyses to various image c g 
schemes and the associated image-restoration and feature-extraction algorithms. 
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Continuous ^ 


Discrete 


Continuous 


Figure 1. Model of image gathering, digital processing, and image reconstruction. 


This work is summarized in references 1 through 6. 

1. F. O. Huck, C. L. Fales, D. J. Jobson, S. K. Park, and R. W. Samms, “Image-plane 
processing of visual information,” Applied Optics, Vol. 23, No. 18, pp. 3160-3167, 1984. 

2. F. O. Huck, C. L. Fales, N. Halyo, R. W. Samms, and K. Stacy, “Image gathering and 
processing: Information and fidelity,” J. Opt. Soc. Am. A2, 1644-1666 (1985). 

3. F. 0. Huck, C. L. Fales, J. A. McCormick, and S. K. Park, “Image-gathering system design 
for information and fidelity,” J. Opt. Soc. Am. A5, 285-299 (1988). 

4. C. L. Fales, F. O. Huck, J. A. McCormick, and S. K. Park, “Wiener restoration of sampled 
image data: End-to-end analysis,” J. Opt. Soc. Am. A5, 300-314 (1988). 

5. J. A. McCormick, R. Alter-Gartenberg, F. O. Huck, “Image gathering and restoration: 
Information and visual quality,” J. Opt. Soc. Am. A6, July (1989). 

6. F. O. Huck, S. John, J. A. McCormick, and R. Narayanswamy, “Image gathering, coding, 
and restoration: Information efficiency and visual quality.” Visual Information Processing, 
Williamsburg, VA, 10-12 May, 1989. 


2. FOCAL-PLANE PROCESSING TECHNIQUES AND TECHNOLOGY 


Image gathering and coding are commonly treated as tasks separate from each other and from 
the digital processing used to restore and enhance images or extract features such as primal sketches 
or contour outlines. However, if we implement the edge enhancement required to draw primal 
sketches with focal-plane processing by properly combining optical response with lateral inhibition 
(as depicted by figs. 2 and 3), then a number of advantages can be gained. These advantages include 
improved resolution by a factor of four, improved robustness to noise, reduced data processing by 
two orders of magnitude, and reduced data transmission by nearly one order of magnitude. The 
implementation of a sensor-array focal-plane processor with lateral inhibition is supported by an 
SBIR contract (NASl-18287, Phase II). If this approach could be extended to include not only edge 
enhancement but also edge detection so that only significant primal sketches need to be transmitted, 
then it would be possible to reduce data transmission by two orders of magnitude without loss of 
the improved accuracy in edge location. 



Lens 



Figure 2. Model of optics and sensor array with focal-plane processing. 
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Optical design, Sensor-array with Combined response 

including lens apodization 3 x 3 lateral-inhibitory processing 


Figure 3. Laplacian-of-Gaussian response created by optical design and lateral-inhibitory piocessmg. 


Our major emphasis is on the development 
intensity adaptive image coding. This work is 
NAS1-18850, both Phase II). 


of an image-coding method that we refer to as local 
supported by two SBIR contracts (NAS1-18664 and 


Local intensity adaptive image coding consists of an innovative model of processing in the human 
retina^ferred^o^ aslntensity Dependent Spread (IDS) and some additional logic to extract contour 
outlines and reflectance ratios at the boundary of two surfaces. Figure 4 is a schematic representation 
of the IDS model. The line of detectors represents a slice through a two-dimensional any 
detectors. When an optical image or light distribution falls on the detector array, then each det 
sends a signal into a network, where it spreads out. Each channel, in turn, sends out a signal that is 
thesum of all the signals that arrive in its location in the summation network. The special property 
of the mS model hi to do with the way the signal from each detector spreads in e — 
network. As depicted in the lower half of figure 4, the magnitude of the signal at its center s 
proportional to the intensity of the light falling on the detector and the spread oi 
inversely proportional to this intensity. The total volume under the spread remains constant. Tha 
is all there is to the model. It has been demonstrated, in reference 8, that this simple space-vanan 
model of image processing has many of the properties of human visual perception. 


iiiimmum 

TTTTTTTTTmrrr 


Detectors 

Summing network 
Output channels 



Detector output pattern: 
Constant volume 


Figure 4. Schematic representation of the IDS model. 


Figure 5 shows the response of the IDS processor to a spot, or point source, that is brighter ^than 
the background and to a step-type edge. Each detector spreads its signal as governed by the intensity 
of the light falling on the detector. For example, all of the spreads for the uniform backgroun in g- 
ure 5(a) are the sfme except for the one detector that is more brightly lighted. Its spread is higher 


narrower. 
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Input 

l(x) 



(a) Impulse response (b) Edge response 


Figure 5. The IDS response to a point source and a step-type edge. 


Each output channel just adds up all of the contributions it receives. The result of this processing 
is shown as the output signal. As can be seen, the IDS response to a point source has a similar 
shape as the response of Marr and Hildreth’s familiar Laplacian of Gaussian (y 2 G) operator for 
enhancing edges (see fig. 2). In fact, the IDS processor exhibits center-surround antagonism and all 
other manifestations of bandpass filtering that have made the \J 2 G operator a favorite algorithm 
for low-level vision processing. However, the IDS response is nonnegative and spatially variant. As 
we will show in the next three figures, the IDS processor accounts for several familiar perceptual 
phenomena of human vision that make it a highly robust low-level vision operator. 


First let us compare the IDS operation to conventional imaging. Figure 6 shows intensity 
profiles taken across conventional and IDS images of a step-type edge input for three illuminations. 
Conventional image-gathering yields a blurred representation that is visually representative of the 
target if the signal-to-noise ratio (SNR) is sufficiently high. As the illumination decreases, the 
representation gets buried in the noise. Image gathering with the IDS processor yields a target 
representation that consists of pulses. The one-crossing of each pulse locates the position of an 
edge in the target. The peak and trough values of the pulse are proportional to the ratio of the 
reflectances at the two sides of the edge, entirely independent of illumination. As the illumination 
decreases, the width of the pulse becomes broader (thereby trading resolution for sensitivity), but 
the accuracy of the one-crossing is unimpaired. For machine vision, this property means that edge 
detection for determining structure is highly robust in widely variant illumination. 


Next let us compare the IDS operation to edge detection with the linear y 2 G operator as well 
as to conventional imaging. Figure 7 shows intensity profiles taken across conventional images and 
outputs from the V l G and IDS operators for two illuminations, high and low. Noise is disregarded 
for simplicity. The peak and trough values of the \7 2 G pulses are proportional to both illumination 
and reflectance. Therefore, it is not possible to characterize the reflectance properties of the target 
independent of illumination. However, the peak and trough values of the IDS pulses are proportional 
only to the reflectance changes. This striking property of the IDS processor mimics human visual 
perception (Weber’s law) and makes it possible to extract the reflectance ratio at the boundary of 
two areas. 
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Conventional imaging 


> DECREASING ILLUMINATION 




IDS operator 



Figure 6. Comparison of IDS operation to conventional imaging. 
Conventional Imaging V^G Operator IDS Operator 


Input 



Figure 7. Comparison of IDS operation to conventional imaging and XJ 2 G operation. 


Figure 8 illustrates a particularly important characteristic of the IDS filter, namely, the robust- 
ness of its reflectance representation to local variation in illumination (e.g., shadow). The recovered 
target 8(d) resembles the original one 8(a) and not the shadowy one 8(b), which is the one that 
was filtered. Traces of the shadow degradation can be seen in the modest loss of accuracy in the 
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actual transition as the illumination decreases. An important extension of IDS processing would be 
to extract color. Color then could be correctly detected independent of illumination. 



spatially varying to target (b) 

illumination 

Figure 8. IDS response to target with spatially varying illumination. 

Odetics, Inc., is now under contract with LaRC (NAS1-18664, Phase II) to develop a hardware 
implementation of the IDS processor (see fig. 9). This processor will be capable of handling image 
data at real-time TV rates (30 frames per second). It will be implemented on several boards for 
the DATACUBE of Sun image-processing work stations. These boards are expected to become 
commercially available in Fall 1989. 



(a) Original target 
with constant 
illumination 



(d) Reflectance 
recovery 



Figure 9. Implementation of IDS for real-time operation in Sun workstation. 

The full potential of the IDS processor for data compression as well as image enhancement and 
feature extraction is realized, of course, only when it is implemented as a focal-plane processor, or 
retinal camera,” depicted in figure 10. The present design of the IDS processor for Sun workstations 
could be implemented in one 5” by 5” board with 8 VLSI chips. A more advanced approach is the 
parallel asynchronous focal-plane image processor depicted in figure 11. This processor, which is 
being developed under a contract with LaRC (NASl-18850, Phase II), is representative of a new 
class of devices that permit full two-dimensional parallel readout and processing perpendicular to 
the focal plane. Advantages over conventional image-gathering and processing techniques include 
rapid parallel distributed processing, high dynamic range, and the elimination of conventional charge 
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transfer, multiplexing, and preamplifiers. Vision processing potentially could be performed several 
orders of magnitude faster than with conventional approaches. Moreover, parallel processing would 
be ideal for tasks like visual pattern recognition. However, the development of this approach is sti 
in its initial experimental stage. 



Figure 10. IDS focal-plane processing camera. 


• 2-D parallel distributed data flow perpendicular to focal plane 
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SI wafer 
Si wafer 
SI wafer 
SI wafer 


• Low power, low noise integrate-and-fire devices for low level 
current-to-frequency conversion in every processing channel 
and at every network mode 


Input 



Figure 11. Parallel asynchronous focal-plane image processor. 
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This work is summarized in references 7 through 12. 

7. R. Alter-Gartenberg, C. L. Fales, F. O. Huck, and J. A. McCormick: “End-to-end 

performance of image gathering and processing for edge detection,” Computer Vision, 
Graphics and Image Processing, submitted. 

8. T. N. Cornsweet and J. I. Yellott, Jr., “Intensity-dependent spatial summation,” 
J. Opt. Soc. Am. 2, pp. 1769-86 (1985). 

9. R. Alter-Gartenberg and F. O. Huck, “Feature extraction from intensity-dependent spatial 
summation.” J. Opt. Soc. Am. A, submitted. 

10. R. Alter-Gartenberg, R. Narayanswamy, and K. S. Nolker, “From primal sketches to 
the recovery of intensity and reflectance representations,” Visual Information Processing, 
Williamsburg, VA, 10-12 May, 1989. 

11. D. D. Coon and A. G. U. Perera, “Integrate-and-fire coding in Hodge-Huxley circuits 
employing silicon diodes,” Solid State Electronics 31, 851, 1988. 

12. D. D. Coon and A. G. U. Perera, “Integrate-and-fire dynamics and spike train information 
coding in neuron equivalent circuits employing silicon diodes,” Neural Networks 1, 1988. 


3. ADAPTIVE IMAGE CODING SYSTEM 


The problem of image gathering and data compression for the high-resolution, high-frame-rate 
video system is such that no single data-compression method, or even just a few methods, can 
satisfy most of the diverse requirements. For some experiments the data-compression requirements 
even change with time. Consequently, we have started to investigate the concept of adaptive image 
coding. This investigation is supported by one SBIR contract (Phase I). We are currently seeking 
additional SBIR contracts under the subtopic 07.01 entitled Focal Plane Image Processing. 

Figure 12 presents a block diagram of the basic adaptive image coding system. The idea of 
this system is to use knowledge-based supervision to autonomously select the appropriate data 
compression scheme based on (a) the properties of the acquired image data (and changes in these 
properties), (b) the visual information required by the investigator, and (c) the available channel 
capacity. The system would have continued access to the investigators’ knowledge of what is 
significant. Under the investigators’ supervision, it could adaptively select image-coding and feature- 
classification schemes based on properties of the scene such as spatial structure, texture, and spectral 
reflectance. 



Figure 12. Schematic depiction of adaptive image coding system. 
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For example, consider figures 13 and 14. In one extreme, if the target of interest is well 
illuminated, as it is for figure 13, and the goal is to reconstruct an image of this target with 
pleasing visual quality, then the appropriate data-compression scheme should allow only those 
degradations to occur that axe benign with regard to visual quality. Furthermore, Optivision s 
OPTIPAC coding system, for which these results axe shown, allows the relationship between visual 
quality and amount of data compression to be continuously adjusted. This adjustment, for example, 
could be autonomously controlled by the channel capacity that is available for transmitting the 
desired images. 



Figure 13. Data compression for well illuminated target. The compression ratio varies from CR = 22:1 to 92:1 for 

the original color pictures shown here only in black and white. (Courtesy of Optivision, Inc.) 

In the other extreme, consider the scene shown in figure 14. An extreme range of light intensity 
exists, stretching from deep shadow to highly reflective surfaces in direct sunlight. It is now important 
to monitor the movements of the astronaut in deep shadow, so an entirely different coding scheme is 
appropriate. The results shown in figure 14 were obtained with the locally adaptive coding method 
based on the IDS model of retinal processing described in Section 2 above. Again, it should be 
possible to transmit only the data required for primal sketches at very high data compressions, or 
more complete data that permits the recovery of reflectance representations. 
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Figure 14. Feature extraction under conditions of extreme variations of light intensity, stretching from deep shadow 
to highly reflective surfaces in direct sunlight. 
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N91-14580 


HIGH RESOLUTION, HIGH FRAME RATE VIDEO TECHNOLOGY DEVELOPMENT 

PLAN AND THE NEAR-TERM SYSTEM CONCEPTUAL DESIGN 

Robert A. Ziemke 
NASA Lewis Research Center 


BACKGROUND 

The obiective of the High Resolution, High Frame Rate Video Technology ™T) 
development effort is to provide technology advancements to remove const 
on the P amount of high speed, detai led optical data recorded 
for microgravity science and applications experiments. These advan ^ents 
will enable the development of video systems capable of high resolut , 9 

frame rate video data recording, processing, and transmission. 

The HHVT users' requirements and near-term technology capabilities have been 
assessed through surveys conducted by NASA Lewis Research Center pace 
Experiments Division from December 1986 through December 987. These 
prel imi nary surveys were designed to form a database to aid in defin ng the 
?ntt!a! direction of the HHVT development effort. The detailed results of 
these surveys are presented elsewhere in this conference paper. 

In summary, the users' requirements survey showed a very wide range of 
requirements for resolution, frame rates, and other Pieters many of which 
are well bevond the capability of today's technology and that of the foreseeaoie 
future. The technology survey showed that most of the available image censors 
offer maximum pixel scan rates in the range of 5 to 15 Mpi xel s/sec/channel , 
while the fastest data storage medium (dynamic RAM) can record at 20 by 
8— b 1 t channel. The largest data storage capacities can be achieved with 
maanetic tape or optical disk, but at the expense of data transfer rate, me 
la?ge disparity that exists between the users' requirements and the near-term 
technology capability makes it obvious that for the foreseeable future .HHVT 
system performance will be driven by the technology capability rather than by 
the user's requirements. 

Three techniques have been identified which will allow efficient use of the 
currently available technology 

(A) Simultaneous multichannel image scan and parallel channel data 

transfer 

(B) Video parameter tradeoff 

(C) Use of dual recording media 

(A) Simultaneous Multichannel Image Scan and Parallel Channel Data Transfe r 

The use of simultaneous multichannel image scan (Fig. ! ) and pa ral lei channe’ 
data transfer allows potentially very high pixel scan rates to be achieved 
compared to single channel scan (Fig. 2). This is true s nee the to P 
scan rate is directly proportional to the number of parallel channels. 

However, this approach has several disadvantages: 
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(1) Each channel requires a separate video amplifier and A/D converter 

(2) The channels must be closely matched, that is, in sensitivity and 
gamma, to avoid striped shading in the displayed image 

(3) Multiple coaxial cables or fiber optic paths are required to connect 
the camera to the central processing unit 

(4) Circuit complexity is increased in proportion to the number of 
parallel channels 

(5) Data word length is proportional to the number of parallel channels 

In view of the advantages and disadvantages cited above, an eight channel 
parallel configuration appears to be a reasonable compromise between speed and 
front-end complexity. At 10 Mpi xel s/sec/channel , the resulting scan rate will 
be 80 Mpi xel s/sec . 

(B) Video Parameter Tradeoff 

More effective use of the 80— Mp 1 xel s/sec scan rate can be achieved by trading 
off video parameters. For example, resolution (pixels/frame) may be traded 
off to obtain a higher frame rate, as shown below: 

80 Mpi xel s/sec 0O _ . 

1024 x 1024 pixels/frame " frames/sec 

80 Mpixel s/sec in col _ . 

64 x 64 pixels/frame = ^ ^ frames/sec 

Similarly, gray scale (bits/pixel) may be traded off to reduce the data 
storage required per frame, or to increase the frame rate and/or pixels/frame 
when the system is not sensor pixel rate limited. The use of video parameter 
tradeoff allows higher resolution or higher frame rate to be achieved, but not 
both simultaneously. 

When trading off resolution (pixels/frame) one of two approaches may be used 
to reduce the resolution. These are illustrated in Figure 3. The first (A) 
increases the pixel size, while maintaining a constant scanned area at the 
sensor target. The second approach (B) maintains constant pixel size, but 
scans a reduced area or a subframe. Subframing (see Fig. 4) is best 
Implemented in an addressable pixel array and offers the following advantages: 

(1) PI xel-per-frame usage is efficient since the size and shape of the 
subframe may be adjusted to cover only the area(s) of interest within the full 
image sensor field of view. 

(2) In an addressable array, the spatial accuracy of the transmitted 
image is based on the relatively high full frame resolution of the sensor, not 
the subframe resolution. 

(3) The pixel scan rate (pixels/sec) can be held constant while the 
system is operating at various subframe sizes and frame rates. This is 
advantageous for tube-type image pickup devices since the beam scanning 
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velocity and the resulting device readout sensitivity would be constant. 

Also, the required beam current would be constant. 

(C) Dual Recording Media 

The use of dual recording media (i.e., dynamic RAM together with a large 
capacity magnetic tape recorder) allows the achievement of both high data 
acquisition rate and large volume data storage. Here, video imagery can be 
captured in dynamic RAM at a very high rate and then transferred to magnetic 
tape at a lower rate. The process can be repeated until the mag tape is 
filled to capacity. The relatively small storage capacity of the dynamic RAM 
limits the amount of video that can be recorded per run. 

TECHNOLOGY DEVELOPMENT PLAN 

The approach for accomplishing the objective of the HHVT development effort 
has two parallel development thrusts consisting of a long range 
component/system technology development effort and a shorter range development 
effort focused on delivering a state-of-the-art system capability. The 
shorter range effort will take advantage of commercially available 
components/systems and the capabilities of those components/systems expected 
to emerge from the laboratory over the next few years. The LeRC ATD project 
schedule covering the HHVT activities through CY 1991 is shown in Figure 5. 

(A) Short Range Development Effort 

The objective of the HHVT short range development effort is to develop a 
prototype video system which will take maximum advantage of the limited 
near-term technology. This will be Implemented as an HHVT system designed to 
enable the tradeoff of video parameters in a single, advanced, flexible, ^ 
digital video system, so that full advantage can be taken of today's limited 
pixel scan rates, data transfer rates, and data storage capacity. The short 
range development effort will be divided into two phases. 

For Phase I (based on 1988 technology), a laboratory demonstration breadboard 
system will be developed which can be upgraded as the technology advances. 

The breadboard system will be a monochrome, digital video system which will 
serve as the foundation upon which future HHVT technology can be developed. 

It will be used as a learning tool and will provide an opportunity to design, 
develop, and gain experience with the basic techniques and building blocks 
needed for an advanced HHVT video system. 

The following capabilities will be developed and integrated into the 
breadboard system: 

(1) Technology to record and reproduce high resolution, high frame rate 

video images in dynamic RAM (128 Mbyte) and 1/2-in. rotary head 
magnetic tape (4-Mbyte/sec, 5.2-Gbyte capacity) 

(2) Technology to provide system flexibility (i.e., the ability to trade 

off frame rate, pixels per frame, and gray scale resolution) 

(3) Subframing capability 
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(4) A high resolution pixel addressable camera. For Phase I, this will 

be implemented as a modified plumbicon tube camera with 102‘4 x 1024 
pixels, and a 40 Mpixel/sec scan rate 

(5) A universal digital video data acquisition, storage, and transmission 

format 

(6) Ancillary experiment data and video system control information 

recorded with each frame 

(7) Remote control of video system via digital commands 

(8) Pretrigger on external event 

The Phase II effort (based on 1989-1990 technology) will consist of upgrading 
the Phase I system as follows: 

(1) Replacing the Phase I camera with a high resolution (1024 x 1024 

pixel), 8 parallel channel, solid state, addressable pixel sensor 
and camera head 

(2) Doubling the pixel scan rate to 80 Mpixels/sec 

(3) Quadrupling the dynamic RAM capacity to 512 Mbytes 

(4) Adding a magnetic tape recorder/reproducer built to MI L-STD-2 1 79 

(30-Mbyte/sec transfer rate, 99-Gbyte capacity) 

(B) Long Range Development Effort 

The long range development effort will provide the critical components needed 
to upgrade the near-term systems, and will provide technology advancements 
needed for future higher performance, advanced video systems. As presently 
planned, the long range advanced technology development effort will initially 
address the following: 

(1) Design and development of a solid state, addressable pixel sensor and 

camera head for Phase II of the short range development effort and 
beyond. (This effort will start as soon as possible after the HHVT 
Workshop . ) 

(2) Design of a pixel addressable color (or multichannel spectrally 

selective) camera using three solid state sensors, each with 
interchangeable filters 

(3) Fiber optics to link the camera head to the central processing unit 

(4) Data compression (This effort is already underway.) 

(5) Automatic subframe tracking, where the subframe automatically tracks 

a moving object within the field of view of the sensor. 
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PROPOSED NEAR-TERM SYSTEM 


The hardware built as a result of the short range development effort Phases I 
and II are called "Phase I Near-Term System" and "Phase II Near-Term System 
respectively. A basic block diagram of a flight oriented HHVT video system 
with dual recording media and downlinking capability is shown in Figure 6. 

This represents the basic configuration of the proposed near-term HHVT systems. 

(A) Phase I Near-Term System 

Figure 7 shows the Phase I Near-Term System block diagram. This configuration 
will be implemented initially as the breadboard system. The video signal from 
the camera head is digitized and demultiplexed into eight digital channels, 

8 bits each, with sync and ancillary data interleaved with the video. The 
composite data is then parallel transferred via the 64-bit high speed videw 
data bus to the 128-Mbyte high speed RAM or the digital magnetic tape. Data 
transfers between runs are made from RAM to magnetic tape via the high speed 
data bus and the video data MUX. 

The VME bus is used to transmit control commands to the subsystems, to 
transmit ancillary data to the high speed video interface for insertion into 
the video data stream, and to transfer low rate video from RAM or mag tape to 
the communications interface for downlinking. The video system can be 
remotely controlled via uplink commands or commands from the payload 
specialist console. The camera control commands are shown in Table 1. The 
ancillary data to be stored with each frame is shown in Table 2. 

Plumbicon Camera 


The breadboard system camera head uses a 1.5 in. electrostatic deflection 
plumbicon tube as the image sensor. The electrostatic plumbicon tube was 
selected for the breadboard system because it is readily available, has 
sufficient MTF (60%) at 1024 x 1024 pixels, can scan at 40 Mpixels/sec, and 
can be operated in a pixel addressable mode. An addressable solid state 
sensor with sufficient resolution and scanning speed is not presently 
available and would have to be developed. 

The plumbicon camera head will be used as a development tool and is planned to 
be used only with the breadboard system to permit the development and checkout 
of the other HHVT system components. A separate solid state sensor/camera 
development effort will be started early so that a suitable solid state camera 
will be available for Phase II and ultimately a flight HHVT video system. 

The plumbicon is also relatively immune to raster burn. This would normally 
be a problem with other pickup tubes when returning to full scan after 
operating in a subframe mode for an extended period of time. 

It is proposed that the plumbicon camera optics consist of the following: 

(1) Interchangeable Objective Lens (F mount 35-mm camera format) 

(2) Removable 40 mm MCP gated intensifier (40 mm image diagonal) with a 

relay lens (1.43 image reduction ratio) 
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A digitally controlled pixel addressable electrostatic deflection system Is 
shown In Figure 8. The subframe start and stop coordinates are loaded from 
the VME bus Into the control registers. An external frame start trigger pulse 
Initiates the scanning of each frame. 

The horizontal sweep sawtooth waveform Is obtained from a 0/A converter which 
Is driven by a binary counter. The counter Is advanced one count on each 
pixel clock cycle. Retrace occurs when the count equals the binary number 
stored In the "horizontal stop address" control register. The horizontal 
starting position of each line is controlled by the binary number stored In 
the "horizontal start address" which is used to preset the counter. The 
vertical scan circuitry operates in a similar manner. The vertical binary 
counter is advanced one count during each horizontal retrace interval. 

Video Image Acquisition Cycle 

Figure 9 shows the video Image acquisition cycle concept which is a unique 
design feature of the HHVT near-term system. One frame is scanned per 
acquisition cycle. Each cycle or frame is initiated by a trigger pulse from 
the frame rate pulse generator (see Fig. 7). 

The frame rate pulse generator is essentially a programmable frequency divider 
clocked by the crystal controlled pixel clock. The frequency divider ratio, 
and hence the frame rate, Is remotely controlled. The principal advantage of 
the triggered acquisition cycle approach Is that the frame rate can be very 
precisely set to a wide range of convenient values such as a 100, 200, 500, 
1000, 2000, 5000, etc. frames/sec. This is very desirable when the HHVT video 
system is used for motion analysis. 

Pretriqqer on External Event 

The use of RAM as a video storage medium permits the video system to be 
operated In a standby recording mode where the "first in" video data stored in 
RAM is continuously overwritten with current video data. When the external 
trigger event occurs, the recording process is continued for a predetermined 
number of frames and then is stopped. Contained in RAM is a continuous 
sequence of video frames representing time before, during, and after the 
trigger event. 


This mode of operation is useful when attempting to capture image sequences of 
unpredictable and sporadically occurring experimental phenomena. The images 
can be recorded without consuming enormous amounts of video data storage while 
waiting for the event to occur. 

(B) Phase II Near-Term System 

Figure 10 shows a block diagram of the Phase II near-term system. This is 
essentially the Phase I system upgraded with a new camera containing an eight 
channel custom 1024 x 1024 pixel solid state image sensor. The upgrade also 
Includes more dynamic RAM and the MIL-STD-2 1 79 mag tape recorder with its 
higher data transfer rate and larger storage capacity. The upgraded system 
has eight parallel video channels from the sensor all the way through to the 
64-bit high speed data bus. The Phase II system scan rate will be at least 
80 Mpixels/sec. 


64 



The proposed Phase II system physically will consist of four packages to house 
the components shown In Figure 10. 

1. Camera head (dimensions to be determined) 

2. Chassis (19 in. wide by 14 in. high by 25 in. deep) 

3. VME bus chassis (19 in wide by 14 in. high by 25 in. deep) containing 

the fol lowi ng: 


A. 

B. 

C. 

D. 

E. 


VME BUS backplane 

System controller boards (CPU, RAM, ROM) 

Mag tape controller board _ 

Data acquisition and ancillary information encoder boards 
Communications interface board 


4. MI L-STD-2 1 79 mag tape recorder (16 in. wide by 17 in. high by 16 in. 
deep) with record/reproduce electronics 

Figure 11 shows the pixel layout of the solid state image sensor with its 
eight parallel channels and the 8 x 8 pixel addressable blocks. 

Theoretical Performance of the Phase I and II Near-Term System s 

Tables 3 to 6 show the theoretical performance of the Phase I and Phase II 
systems at various subframe sizes and gray scale levels when recordi ng 1 n 
dynamic RAH and magnetic tape. The resulting frame rates do Rot include the 
effects of the horizontal and vertical retrace blanking Intervals. The actual 
frame rates will be somewhat lower. The increased frame rate and the 
increased total frame storage capacity which result when pixels per frame are 
traded off are readily apparent. Trading off gray scale increases the frame 
storage capacity. However, trading off gray scale will not increase the rame 
rate once the maximum sensor pixel scan rate is reached. 

Figures 12 to 15 show how the Phase I and Phase II system performance compares 
to the users' requirements on a data acquisition rate basis and on a data 
storage capacity basis. The performance of each system (i^e. data 
acquisition rate in Mbyte/sec or data storage capacity in Mbytes or Gbytes) is 
represented by a diagonal line superimposed on the . u s er s' requirements^ In 
aeneral the users' requirements represented by points below and to the left 
of the diagonal line can be met by that system. One byte/pixel is assumed for 

these compar i sons . 

Sync and And 1 1 ary Data 

When scanning and digitizing an image to generate video data, sync information 
must be periodically inserted in the data stream so that the image can be 
later reconstructed. For scientific video imagery it is also desirable to 
periodically insert other ancillary data. See Table 2. 

The designer has the choice of inserting the sync and ancillary (S& A> d *ta 
between each full video frame or placing the S&A data within the video frame 
where some of the pixel space is occupied by S&A data. There are two 
advantages to placing the S&A data within the frame. 
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1. Each frame together with its S&A data always occupies memory in blocks 

where the number of bits per block is an integral power of two. Such 

data blocks stack up neatly in disk and tape data storage. For 
example, a 256 x 256 pixel frame at 4 bits/pixel with S&A data within 
the video frame occupies 32 768 bytes or exactly 64 disk sectors at 
512 bytes/sector . If the 512 S&A data bits are inserted between 
frames, each frame with S&A data occupies a total of 32 832 bytes or 
64.125 di sk sectors . 

2. With S&A data within the frame, the sync data can be further divided 

into. frame sync and line sync. The use of line sync results in faster 

initial synchronization and faster resynchronization in the event of 
momentary transmission loss or sync data errors. 

It is recommended that the near-term HHVT systems be designed to place the S&A 
data within each frame as shown in Figure 16. Each frame starts with a 16 bit 
frame sync word, followed by a 16 bit line sync word occurring at the 
beginning of each group of eight lines. The ancillary data (384 bits total) 
consists of eight groups of 48 bits interleaved among the first eight sync 
words (one frame sync plus seven line sync words). The frame sync word is 
shown shaded. 

The picture information lost due to the S&A data contained within a 64 x 64 x 1 
frame (64 x 64 pixels, 1 bit/pixel) is illustrated in Figure 16. In this frame 
the S&A data occupies 512 bits. Figure 17 shows the lost picture information 
in a 128 x 128 x 1 frame. Note that when the pixels per frame is quadrupled 
the lost picture area is effectively cut in half. As the bits/pixel is 
Increased, the number of lost pixels will proportionately decrease. The 
vertical column at the left containing the S&A data is blanked off on the 
displayed image. For the 128 x 128 x 1 frame the actual displayed picture 
area is 120 x 128 pixels. 

In playing back a sequence of frames, it is envisioned that once the composite 
video data is received and stored on disk or magnetic tape, the display 
processing software will look for three or more consecutive line sync words by 
means of a repetitive sync window. The line sync words are interleaved with 
the video data at a known spacing or repetitive pattern where the spacing 
depends on the specific pixels/frame and bits/pixel. Once the software locks 
onto the line sync repetition pattern, it will step through the line sync 
words and eventually find the next frame sync word. In essence, the line sync 
leads the system to find the frame sync. Once the frame sync word is found 
synchronization has been achieved. 

It Is necessary that the system look for a repetitive sync pattern before 
attempting to synchronize, since the sync words are not unique from the video 
data. It. is possible from time to time to have consecutive pixels form the 
same 16 bit word as the line or frame sync word. Looking through a sync 
window for a repetitive sync pattern and locking onto the pattern makes the 
system immune to these spurious sync words. 

Video Data Format 


Figure 18 is a graphic representation of the high speed video RAM. Shown on 
the left are the 64 parallel data bits from the high speed video data bus. 
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video channels 
into RAM In 64 


at eight bits per channel, 
bit parallel words starting 


The 

at 


. In 
sync 
contai n 
bits) 
video 


These are divided into eight 
data is sequentially clocked 
address 0000000. 

Figure 19 shows how the video, sync, and anci 1 lary data wil i ap P®J r R ^,f or 
the Phase II system. With the data organized in this way, the S&A data will 
occupy the left edge of the video frame as shown in Figures 16 and 17. 

Fiqure 19 the first 64-bit word (address 0000000) contains the frame 
(16 bi ts) and 48 bits of ancillary data. The next seven 64-bit words 
the first eight lines of video data. Next is the ® 

and 48 more bits of ancillary data, followed by seven 64-bit words of 
for lines 9 through 16, and so on. 

This is a flexible format that can accommodate various pixels/frame (needed 
for subframing) and various gray scale resolutions (bits/pixel ) . In ^ ea Jl n L A 
either simply increases the amount of video data placed between the 64-bit S&A 
words In Figure 19 the spacing between the S&A words will increase, as will 
the total number of 64-bit video data words required per frame. 

The memory allocation for each of the eight parallel channel s as a function of 
bits/pixel is shown graphically at the bottom of Figure 19. Each of the four 
blocks represents the first line (64 pixels) of the raster shown In Figure 16. 
The first eight bits (shown shaded) of each block are part of the 16-bit frame 
sync word. Note that the pixels are numbered in Figures 16 and 19. 

Figure 20 shows graphically the data organization in RAM for the Phase I 
system. This organization is different from the Phase II system data 
organization since the plumbicon is a single channel device. This 
organization is required so that the S&A data will occupy the left edge o 
video frame as shown in Figures 15 and 16. 


the 


The plumbicon's single video output, unlike the eight channel solid state 
sensor, is demultiplexed sequentially to the eight digital channels. The 
result is that each scan line is distributed among the eight channels. The 
S&A data is now all contained in channel 1. 

The memory allocation as a function of bits/pixel is shown to the right of 
Figure 20 Again, each of the four blocks represents the first line (64 pixels) 
of the raster shown in Figure 16. 

SUMMARY 

The objective of the High Resolution, High Frame Rate Video Technology (HHVT) 
development effort is to provide technology advancements to remove constraints 
on the amount of high speed, detailed optical data recorded and transmitted 
for microgravity science and applications experiments. These advancements 
will enable the development of video systems capable of high resolution, hig 
frame rate video data recording, processing, and transmission. 

The results of the initial surveys show a large disparity between the 
microgravity science and applications video users' requirements and the 
near-term technology capability. The initial objective of the HH VT 
development effort is to develop techniques to allow the most efficient use of 
the limited capability of the current and near-term technology so that we may 
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begin to close the gap between the users' requirements and the technology 
capability. Techniques such as multichannel image scan, video parameter 
tradeoff, and the use of dual recording media have been identified as methods 
of making the most efficient use of the near-term technology. 

A technology development plan has been formulated which has two parallel 
development thrusts consisting of a long range component/system technology 
development effort and a shorter range effort that will take advantage of 
commercially available components expected to be available in the near 
future. The short range effort will focus on developing the techniques 
mentioned above to make the most efficient use of the near-term technology. 
Emphasis will be on system flexibility, that is, the ability to trade off 
resolution, frame rate, and gray scale in one system so that today's limited 
pixel scan rates and data transfer rates can be used to full advantage. A 
video data format designed to accommodate the system flexibility is discussed. 

The short range effort will be divided into two phases. Phase I will be the 
development of a laboratory demonstration breadboard system which can be 
upgraded as the technology advances and will serve as the foundation upon 
which future HHVT technology can be developed. As currently planned, the 
Phase I system will Incorporate a pixel addressable plumbicon tube camera. 

Phase II will consist of upgrading the Phase I system with a custom pixel 
addressable multichannel parallel scan solid state camera and other 
Improvements . 

The long range effort will provide the critical components needed to upgrade 
the near-term systems and will provide the technology advancements needed for 
future higher performance, advanced video systems. 
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Table 1 


PAMFRA CONTROL CO MMANDS 


COMMAND 


RTTS REQUIRED 


SCAN START ADDRESS: (X) 

( Y> 

SCAN STOP ADDRESS: (X) 

(Y) 

PEDESTAL LEVEL: 

VIDEO gain: 

INTENSIFIER GATE TIME: 
SENSOR INTEGRATION TIME: 
IRIS OPEN (RATE 1 ,2,3) : 
IRIS CLOSE (RATE 1.2,3) : 
FOCUS IN (RATE 1,2,3): 
FOCUS OUT (RATE 1*2,3): 
ZOOM IN (RATE 1,2,3): 
ZOOM OUT (RATE 1,2,3): 


7 

7 

7 

7 

8 
8 

12 

12 

2 

2 

2 

2 

2 

2 


TOTAL BITS REQUIRED: HU 

TOTAL BITS PLANNED: 128 (16 BYTES) 


Table 2 

AMPTI I ARY INFORMATION S T ORFD WI TH FflCH FRAM E 


PflRflMETEB 

CAMERA NO: (1 THROUGH 8) 

FRAME NO: (0 TO 1,048,576) 

time: day xxx 

HR. XX 

MIN XX 

SEC XX.XXXX 

FRAME RATE: X.XXEXX 

SUBFRAME ORIGIN: (X) 

(Y) 

SUBFRAME DIMENSIONS: (X) 

(Y) 

GRAY SCALE RESOLUTION (BITS/PIXEL): 

16 EVENT MARKERS: 

EXPERIMENT DATA (BCD, BINARY, OR ASCII): 


RTTS REQUIRED 

3 (BINARY) 

20 (BINARY) 

9 (BCD) 

7 (BCD) 

7 (BCD) 

27 (BCD) 

35 (ASCII) 

7 (binary) 
7 (binary) 

7 (binary) 
7 (binary) 

3 (binary) 

16 

64 


TOTAL BITS REQUIRED: 219 

TOTAL BITS AVAILABLE: 384 <40 BYTES) 
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lead CENTER: LeRC ATD PROJECT SCHEDULE 

performing SUPPORT centers: iaRc. msfc HIGH RESOLUTION. HIGH FRAME RATE 
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Down link/Upl ink 
To Ground-Based 
Video Receiving System 








Video System 
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Figure 8 

Pixel Addressable Deflection Concept 
for Electrostatic Plumbicon Camera 
(8X8 Pixel Blocks) 
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ECL Component 









Video Image Acquisition Cycle 
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Intenaifier Gate 


Phase II Near-Term HHVT Video System 
(Flight System Block Diagram) 
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Near-Term System Performance 
Data Acquisition Rate vs User Requirements 
Mode: Continuous Tape Record 



(•oss/'Jj) 31Vd 


82 


ORIGINAL PAGE IS 
OF POOR QUALITY 


RESOLUTION (Pixels/Frame) * Exp. below scale-302, 307, 41 3, 41 5, 41 6, 427a, 50 1,504, 

♦ 1 Byte/pixel 



Near-Term System Performance 
Data Storage Capacity vs User Requirements 

Mode: Burst Record (RAM) * Exp. above scale— 416 

- Refer to appendix / 





Figure 15 

Near-Term System Performance 
Data Storage Capacity vs User Requirements 
Mode: Continuous Tape Record 
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Image Area Occupied by Sync 
and Ancillary Data 



Image Area Occupied by Sync 
and Ancillary Data 
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Figure 18 
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Proposed HHVT Data 
8 Channel Parallel 
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COMMENTS FROM THE USERS WORKING GROUP 

The following summarizes the comments that were leader 

of 6 thesis cussi on, °!]ack Tlalzma", "chief of NASA Lewis Microgravity Science 
and Technology Branch. 

Hhen asked whether the High Resolution, High Frame Rate Video Technology 

(HHVT) project appeared to have a rea1 nants 6 ? n^he Users Working Group was 
requirements, the concensus of the part c pants in the User wo 9 

that it did not. The requirements for color and reso requcst ed 

i they 

needed more than two cameras for , t J?' r .liS'nf the subject. Whoever wrote the 

STlS&.tEd sSme of the questions on the HHVT users' 

questlonnal re . 

, . wflca 1 c cnrvev of users' requirements, the requirements 
MlnSrHlM ted in th HhV^users^requirement^^ 

to‘the S questionnaf r^have^expertments'^that are in the conceptual design phase 

already. 

One participant recommended that NASA enga ge a ^es^g^engineers . 

marketing role of Interacting betwee th ^ g u ^^and the^des^^ ^ ^ hhvt 
S everal participants commented that the The HHVT pro j ect 

Users Workshop without knowing the real needs of the xperi menters and 

team was advised to go out on a one-to-on basis ^tne^^ ^ ^ tQ 

SSIVSTi™'"- Sr«“« •' »" “£ 

ESS ™ 

^Ther^should ie £ ST/^i cation between the users and 
the design engineers. 

One participant expressed the 

few million on HHVT development jjjj the v^j ndustr^' '*5^ experimenters 

S5 deve?op?ng a system Capable of spectral r * dl a 3 C 5 0 ?i3 S thS^KsA 0 " S°Sot 
constrained 1 by IT-StS? "KS ^l^n/high frame 
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rate video technology for space applications is very much needed, and acaaemid 
and industry expect NASA to be the agency pursuing this area of technology 
development. It is unlikely that industry would invest in this area of 
research and development independently. The commercial demand is almost 
nonexistent. One person asked whether NASA actually intends to fund the 
development of imaging chips considering the high cost involved. 

The purpose of the near-term HHVT system is to function as a breadboard to 
test the technology. The participants were asked whether they thought 
Developing the breadboard was sensible. One participant commented that the 
breadboard method is the only logical approach for this type of technology 
advancement. But he cautioned the HHVT project team not to regard the 
breadboard as a flight prototype. Another participant observed that no one 
system couldmeet all of the users' requirements. A more flexible type of 
system with interchangeable components will be needed. NASA will have a good 
system when NASA builds one that satisfies 80 percent of the users' 
requirements, after those requirements have been better defined. 

To some participants the idea of using a plumbicon imager for the Phase I HHVT 
system seemed inappropriate. They advised against using tube cameras in a 
flight prototype of the HHVT system. They recommended that more emphasis be 
p aced on the development of a solid state sensor and more thought given to 
fast CCD readout. The near-term HHVT system should have at least three 
diiferent types of cameras for ultraviolet, infrared, and visible light 
imaging. The rest of the system must be designed to accommodate all of these 
cameras, one at a time. According to the participants, there is no 
requirement identified for true color. False color imaging, achievable with 
the use of trim filters, will suffice. But there is a need for the HHVT 
system to enable spectral radiance measurements. 

The participants were asked whether a data compression capability should be an 
integral part of the HHVT system. For some, data compression will be user and 
image dependent, and should be implemented by the user as he sees fit It 
should not be an integral part of the HHVT system. Other participants 
disagreed, but they pointed out that users would want to be able to control 
the application of compression to their image data. One person observed that 
data compression is a TDRSS-driven problem. Whenever the shortage of downlink 
channels is alleviated, the need for data compression will dissipate. The 
HHVT project team was advised to solicit information on the planned in-space 
operations for each experiment, so as to be able to assess whether real-time 
downlink for a given experiment is a necessity. The participants thought the 
cyclic buffering capability of the HHVT system would be useful. 

When asked what they thought would happen if the HHVT program were 
discontinued, the participants expressed the opinion that the users would all 
build their own video systems tailored to their specific needs. Then someone 
somewhere along the way, would come up with the idea of building one system 
naving all the capabilities of the individual video systems. The participants 
applauded NASA for initiating this High Resolution, High Frame Rate Video 
Technology development project and encouraged the project team to continue 
with the development of the near-term HHVT systems as breadboards 
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for 

The 


COMMENTS FROM THE TECHNICAL EXPERTS WORKING GROUP 

The following summarizes the comments that were "J|^ e A by B *j!^hart t ' CM NASA 
the Technical Experts Working Group meeting. James A. Burkhart,^ 

Lewis Instrumentation and Data Systems Branch, led t 

One of the technical experts advised Jhe High Resolution High 
Video Technology development project team to consider tne s g , 

haw higher' quantum'effkiency, lower nol in-scen^dynamic 
better response uniformity, no signal lag, and much higner 7 

range . 

Another participant questioned the luminosity of the 

in some of the experiments. He ^ndered whet her the use m ve 

The^a^e y a s canjn,y ent 

» *2™ to Interpret' an .«£" ^ve"ture 

^ ?™fe5l te ?^ab,fless than f5 per^enl of the users wh responde^to the 

HHVT r users^' survey S did' not°prov1de°enough Information for sensor manufacturers 
rlnderstand thlMging requirements of the e»per menters^ For each 

experiment they would need to know the sourc distribution. Knowing how 

frame rate, dynamic range requirement, and spectral JJJtnDuti^ ^ 

an experimenter views his final images, a experts^urther advised 

s isTpS:^ r-n. 

s M^rissifi: - tead 

preslnted^by^the^users^nto engineering requirement s^^^ 

and the users. 

commercial systems that could be used. 

in closing, the observation was made that the ^'^ravi ty science^communi ty^^ 
has set forth requirements to sti mul ate video t satisfy all the 

requirements are for a general purpose s vste . ^ f them _ y^e HHVT 

users, the HHVT project team may not be ^' e r !thlt the near-term HHVT system 
^^.SiSS'tolliufri ^^ra?n:2 e serof their stated requirements. 
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FINDINGS AND RECOMMENDATIONS 


Jf t fL t ESw? SerS aPd the . techmcal experts had separately discussed the results 
of H ^ V I users requirements survey, the proposed near-term HHVT system 
and the future course of the HHVT development project, both groups reconvened 
to share their findings and recommendations. The following list of findinqs 
was mutually agreed upon by the users and the technical experts: 


(1) The HHVT project does not have a 
requirements . 


realistic assessment of users 


(a) More information is required on each specific experiment. 

(b) The requirements should be expressed as scientific requirements 
rather than as engineering requirements. 


(2) The Phase I system should be pursued as a breadboard system 
as an operational (flight prototype) system. 


but not 


• [ 3) I he Ph ? se 1 and Phase 11 HHVT ^^ems should be designed to accept a 
variety of modular subsystems as the technology advances. 

(4) True color is not a requirement, but the ability to make spectral 
radiance measurements is. This will require a 12 bit system (an 8 bit system 
is currently planned for both the Phase I and Phase II systems). 

(5) The "burst mode" is highly desirable. 


(6) Onboard viewing is highly desirable. 


_ (7) It is premature to decide whether subframing is preferable to the use 
of integrated pixels. Therefore, both methods should be developed. 

(8) Edge definition processing should be accorded a high priority. 

The following list of recommendations was mutually agreeable to the users and 
the technical experts: 


(1) Each of the investigators should be contacted on an individual basis 
in order to obtain more detailed information on the scientific imaqina 
requirements. 


. . 0nc ® the engineering requirements for the HHVT systems are defined 

the HHVT project team should follow up with the investigators to discuss 
possible tradeoffs. 


(3) In proceeding with the Phase I breadboard system, the HHVT project 
team should make use of their updated survey requi rements . Also, they should 
keep the system flexible and involve the user in system testing and evaluation. 

(4) The HHVT project team should concentrate on developing areas of video 

technology that industry is not developing. One example is flexible wavs of 
compressing data. J 
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ABBREVIATIONS AND ACRONYMS 


ATD 

advanced technology development 

ATDRSS 

Advanced Tracking Data Relay Satellite System 

BER 

bit error rate 

bit 

binary digit 

byte 

eight bits 

CCD 

charge coupled device 

CID 

charge injection drive 

G/T 

gain-to-noise temperature ratio 

Gbit 

Gigabit 

GSTDN 

Ground Spacecraft Tracking and Data Network 

IDS 

intensity dependent spread 

MCP 

multichannel plate 

MOS 

metal oxide semiconductor 

MTF 

modulation transfer function 

MUX 

multiplexer 

NPDT 

nonparallel (magnetic) disk transfer 

PDT 

parallel (magnetic) disk transfer 

pel s 

picture elements 

pixels 

picture elements 

RAM 

random access memory 

RGB 

red, green, blue 

SBIR 

Small Business Innovative Research 

SCSI 

small computer system interface 

SFR 

spatial frequency response 

TDRSS 

Tracking Data Relay Satellite System 

VLSI 

very large scale integration 

YIQ 

luminance (Y) and chrominance (I,Q) signals 
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